Performance Test Plan for <Application Name>
Performance Center of Excellence(PCoE)
TMH
7/10/2014

Contents

1.0	Introduction	3

1.1	Purpose	3

1.2	Distribution List	3

1.3	Document History	3

2.0	Executive Summary	3

3.0	Goals	3

4.0	Scope	3

4.1	Items in Scope	3

4.2	Items Out of Scope	3

5.0	Assumptions and Dependencies	3

6.0	Performance Metric	3

7.0	Environment Specifications	3

7.1	Performance Test Environment:	3

7.2	Production Environment:	3

8.0	Performance Test Phases	3

9.0	Entry & Exit Criteria	3

9.1	Entry Criteria	3

9.2	Exit or Acceptance Criteria	3

10.0	Test Execution Management	3

10.1	Test Data	3

10.2	Test Execution Details	3

10.3	Test Scenario Design	3

10.4	Test Schedule	3

10.5	Roles and Responsibilities	3

11.0	Defect Management Process	3

11.1	Issues & Risks	3

11.2	Open Items	3

12.0	Test Tools and Analysis	3

12.1	Test Execution	3

12.2	Test Monitoring	3

12.3	Test Analysis	3

13.0	Appendix	3

14.0	Glossary	3

Introduction

Purpose

The Performance Test Plan document is a key deliverable of the Performance Test team. This document will serve as a roadmap detailing all of the activites/phases of the performance testing cycle . This document describes in detail the scope of involvement of performance testing team along with the tasks and actions that will be carried out from initiation through closure of the performance test projects.

Distribution List

The intended audience is as follows:

Solutions Architect

Application Owners

PCoE Test Manager

PCoE Team

Document History

Table 1: Document History

Version	Date	Author	Revision History
0.1	10-Jul-2014	Srinath Mohankumar	Initial Version
0.2	10-Jul-2014	Saleem Khan	Updates made to Test Execution management section

Table 2: Reviewers

Version	Date	Author	Revision History
0.2	07/24/2014	Shanafelt, Garrett
0.2	07/25/2014	Martin, Patrick

Table 3: Approvers

Version	Date	Author	Revision History
0.2	07/25/2014	Shanafelt, Garrett
0.2	07/25/2014	Martin, Patrick

Executive Summary

The Performance Test Plan for <PROJECT NAME> application contains the following information:

Establishment of performance test goals

List of items that are in scope and out of scope

List of assumptions and dependencies

Detailed description of test phases and test cycles

Specific transactions that will be tested

List of performance metrics to be measured and the expected criteria

Detailed schematics of test environment and hardware specifications

Test execution management

Defect management process

Test tools and analysis

Goals

The primary goals of this performance testing effort are mentioned below :

Provide an indication of how the application will support business volumes by obtaining application performance measurements based on current and projected business volumes

Identify and validate performance requirements

Identify application performance issues

Identify the capacity of the application, measured by the maximum number of transactions an application can execute for a given time period

Provide test results that indicate how an application meets the identified performance requirements, including risks and assumptions associated with performance tests

Provide test results to application teams for analysis and scalability assessments

Produce repeatable processes for performance tests.

<Application specific goals>

Scope

Items in Scope

The purpose of this performance test is to make sure the application meets specific business SLAs and requirements, including transaction response time, throughput, and CPU utilization.

Verify that changes to the application, server or workstation software infrastructure will not impact the current production environment by conducting Load, Stress, and Endurance tests.

Provide test results that indicate how an application meets the identified performance requirements including risks and assumptions associated with performance tests.

Provide test results to application teams and all necessary stakeholders for analysis and scalability assessments.

Performance testing of the following business functions:

<Function 1>

<Function 2>

<Function 3>

<OTHER ITEMS IN SCOPE>

Items Out of Scope

Functional testing

Those transactions that have been identified as having low business risk and low performance impact relative to other transactions

<Application Specific out of scope details>

Assumptions and Dependencies

There are assumptions made for this performance testing effort:

Any project scope or requirements changes will follow the Project Change Control Process

If dates are moved forward, the scope of the testing may be reduced, and/or additional resources may be required to complete the tasks in the required timeframe

<OTHER ASSUMPTIONS AND DEPENDENCIES>

Performance Metric

The following metrics will be recorded and reported on in the Performance Test Results Summary for Duration and Load Testing.

Requirment	Success Criteria
System Specific Requirements
% CPU Utilization	Average % of CPU utilized during the test SLA: <value>
Pages/second (Windows Only)	Indicator of physical memory shortfall SLA for Application Server: <value> SLA for Web Server: <value> SLA for DB Server: <value> SLA for <any additional servers>:
Processor Queue Length (Windows Only)	Amount of work waiting for the processor to complete other work SLA for Application Server: <value> SLA for Web Server: <value> SLA for DB Server: <value> SLA for <any additional servers>:
Load Average (UNIX Only)	Indicator of physical memory shortfall SLA for Application Server: <value> SLA for Web Server: <value> SLA for DB Server: <value> SLA for <any additional servers>:<value>
Project Specific Requirements
Throughput	Average throughput projected target, based on projected utilization SLA for Load: <value> SLA for 18-Month: <value>
Transaction Response Time	SLA: <value>
Concurrent Users	SLA: <Number of concurrent Users>

NOTES:

CPU Utilization should be 60% if disaster recovery is setup for a one to one failover

CPU Utilization should be 50% if disaster recovery is not setup for a one to one failover

Pages per second (Windows Only) SLA should be 2 times number of CPUs

Processor Queue Length (Windows Only) SLA should be 2 times number of CPUs

Load Average (UNIX Only) SLA should be 2 times number of CPUs

Throughput will either come from the business person on the project or from current production data

Transaction Response Time will either come from the business person on the project or from current production data

Environment Specifications

Performance Test Environment:

The chart below provides detailed information about test systems to be used for all performance test execution. This information is vital in understanding and interpreting the test results and is a crucial input into capacity modeling exercises.

Server Function	Hardware Specifications	Software Specifications	FQDN	IP Address	Login
Load Generation Systems (LGS)
<CONTROLLER>	<HARDWARE INFO>	<OS TYPE> <ANY OTHER NECESSARY S/W>	<SERVER ADDRESS>	<IP ADDRESS>	<LOGIN>
<GENERATOR>	<HARDWARE INFO>	<OS TYPE> <ANY OTHER NECESSARY S/W>	<SERVER ADDRESS>	<IP ADDRESS>	<LOGIN>
Systems Under Test (SUTs)
<APP SERVER>	<HARDWARE INFO>	<OS TYPE> <ANY OTHER NECESSARY S/W>	<SERVER ADDRESS>	<IP ADDRESS>	<LOGIN>
<WEB SERVER>	<HARDWARE INFO>	<OS TYPE> <ANY OTHER NECESSARY S/W>	<SERVER ADDRESS>	<IP ADDRESS>	<LOGIN>
<DB SERVER>	<HARDWARE INFO>	<OS TYPE> <ANY OTHER NECESSARY S/W>	<SERVER ADDRESS>	<IP ADDRESS>	<LOGIN>

Production Environment:

The chart below provides detailed information about the Production environment configuration. This information is vital in validating test results and is a crucial input into capacity modeling exercises.

Server Function	Hardware Specifications	Software Specifications	FQDN	IP Address	Login
<APP SERVER>	<HARDWARE INFO>	<OS TYPE> <ANY OTHER NECESSARY S/W>	<SERVER ADDRESS>	<IP ADDRESS>	<LOGIN>
<WEB SERVER>	<HARDWARE INFO>	<OS TYPE> <ANY OTHER NECESSARY S/W>	<SERVER ADDRESS>	<IP ADDRESS>	<LOGIN>
<DB SERVER>	<HARDWARE INFO>	<OS TYPE> <ANY OTHER NECESSARY S/W>	<SERVER ADDRESS>	<IP ADDRESS>	<LOGIN>

<This sections provides a high level information for Production Hardware. Refer SAD for details>

Performance Test Phases

The following section summarizes the approach that will be used to conduct performance testing for the <PROJECT NAME> application.

Test Phases

The Performance test will be broken down into four phases: Planning, Preparation, Execution and Reporting/Closure.

The Planning phase of Performance test develops the specific approach and plan that will be used to organize and execute the Performance test

The Preparation phase of Performance test prepares test data, Performance test scripts, and the test environment; assesses requirements for Capacity Modeling for the execution phase

The Execution phase of Performance test executes the performance test scripts created in the preparation phase. Performance test execution will produce results based on stress, load, duration, and capacity tests

The Reporting/Closure of Performance test consolidates and presents the test results generated during test execution. The results will be provided to the application team and applicable stakeholders for analysis.

Entry & Exit Criteria

Entry Criteria

Project Related:

PCoE team engaged during project initiation phase by initiating a performance testing request.

Non Functional (Performance) Requirements and clearly defined SLAs defined by business and reviewed, approved by the technical owner or architect.

Performance Test Questionnaire completed by the application team (wherever applicable)

Test Preparation and Execution:

All of the planned rounds of SIT (Functional testing) are complete with no major or Critical outstanding defects.

Functionally stable application is available and application code frozen before performance test execution.

A dedicated Performance test environment is available, configured and ready for testing.

Tasks

Project Related:

Share effort estimation.

Preparation and signoff - Performance Test Approach and Performance Test Plan

Test Preparation and Execution:

Performance test data setup covering all the performance test cycles.

Setup/configure Performance testing tools and application monitoring counters.

Test script creation, scenario design for performance testing and shakedown

Execute following tests (as applicable):

Load Test

Stress Test

Endurance Test

Result analysis, identify performance issue and defect reporting

Prepare test results summary and closure reports.

Verification and Validation Criteria

Review and analysis of the performance test results against the defined SLAs.

Re-test after the reported Performance test defects are fixed.

Exit or Acceptance Criteria

All of the planned performance test cycles and activities are completed.

All the NFRs are meeting the SLAs.

Application stakeholder reviews and approves the Performance Test results.

Application stakeholders approves the Performance Test Closure Reports

Test Execution Management

Test Data

<Test Data Details > Refer Performance Test Data Requirement document

Test Execution Details

The following data will be used during Load and Duration testing.

# of scripts to test = <# SCRIPTS>

# of Vusers = <NUMBER OF CONCURRENT USERS>

The following scripts will be tested. More details to be mentioned in the Appendix section

<This sections provides high level information for each test. Refer to Performance Test Execution Pan>

Script Name	Desired % of load test activity	Number of virtual users	Response Time Service Level Agreement (SLA)
<SCRIPT #1>	XX%	XX
<SCRIPT #2>	XX%	XX
<SCRIPT #3>	XX%	XX
<SCRIPT #4>	XX%	XX
<SCRIPT #5>	XX%	XX
Total	100%

Test Scenario Design

The Test scenario design will describe the iteration count ,data volume and user concurrency details associated with each script which is aligned to the Test Exection scenario and goals of the test. Please find below the link to the template:

http://teamwork/sites/perftest/Shared%20Documents/Project%20Documents/PCoE/WIP/2.8.4-FN_Test_Scenario_Design_Application_Name_0.1.xlsx

Test Schedule

The test schedule contains key milestone tasks involved in every performance test. Additional tasks may be added to any phase (Test Planning, Test Preparation, and Test Execution) as needed to reflect each project’s individual requirements.

The Test Execution Plan timelines should be in sync with the test schedule .

<Refer Performance Test Execution Plan>

Task	Planned Completion Date	Status	Responsibility
Test Planning
Prepare Initial Test Approach Document	<PERFORMANCE LEAD>
Provide additional input	<PROJECT MANAGER>, <DEVELOPMENT TEAM>, <BUSINESS >, <SA>
Prepare Test Plan	<PROJECT LEAD >
Update Test Plan	<PERFORMANCE LEAD>
Test Plan Signoff	<PROJECT MANAGER>, <DEVELOPMENT TEAM>, <BUSINESS >, <SA>
Test Preparation
Application Walkthrough	<DEVELOPMENT TEAM> <PERFORMANCE ENGINEER>
Determine strategy for load generation systems	< PERFORMANCE ENGINEER>
Create performance test scripts	<PERFORMANCE ENGINEER>
Data collection / parameterization	<DEVELOPMENT TEAM/ PERFORMANCE ENGINEER>
Prepare Execution Plan	<PERFORMANCE LEAD>
Prepare Workload Characteristics	PERFORMANCE LEAD>
LoadRunner Scenario Creation	<PERFORMANCE ENGINEER>
Sample test execution	<PERFORMANCE ENGINEER>
Test Execution
Iteration 1
Load Test	<PERFORMANCE ENGINEER>
Results Analysis	<PERFORMANCE ENGINEER>
Iteration 2
Stress Test	<PERFORMANCE ENGINEER>
Results Analysis	<PERFORMANCE ENGINEER>
Iteration 3
Endurance Test	<PERFORMANCE ENGINEER>
Results Analysis	<PERFORMANCE ENGINEER>
Final Test Report	<PERFORMANCE LEAD >

Roles and Responsibilities

List of roles and responsibilities for all resources involved in the performance test of this application:

Team/Role	Responsibilities
Business	Define Non-Functional requirement Exception Approval for defects Participates in Go-No Go decision tollgate meeting
Technical Owner/ Solution Architect	Review and approve non-functional (performance) requirement Review and approve the test approach, test plan and test closure report Participate during test execution Review the defect, prioritize the defect Facilitate defect resolution Exception Approval for defects Participates in Go-No Go decision tollgate meeting
Project Manager	Request performance testing services with PCoE Facilitate and coordinate overall testing related activities Application walkthrough Sample dataset Test data, etc. Approves test approach and closure report Exception Approval for defects Participates in Go-No Go decision tollgate meeting
Development Team	Review Non-functional requirement Execute Performance Test in development cycle, before deploying the code on QA environment Participate during performance test execution in QA cycle Defect acceptance, identification and fixes
Performance Test Manager	Facilitate PCoE Services Acceptance of performance testing requests Provide effort estimation Review test approach and test plan Participate in test planning and defect discussions Review and sign-off performance test closure report Exception Approval for defects Participates in Go-No Go decision tollgate meeting
Performance Test Team(Performance Architect, Performance Lead, Performance Engineer)	Accept performance testing requests, clarify/request more information on NFRs; work with application/project team for application walkthrough Create Performance test effort estimate Prepare Test Approach, Test plan, BFD, Test scripts, Test scenarios, Test result and Test Closure report Schedule and execute tests Identify and log defects; follow-up and facilitate defect resolution

Defect Management Process

Kindly refer to the Defect Management Workflow document in the below link

<Defect Management Workflow LInk>

Defects identified during performance testing needs to be raised in ALM or any other defect management tool identified for the project.

Issues & Risks

This section calls out any specific risks and issues that present the potential for failure to meet requirements on schedule. Additional items should be added as concerns arise. To manage risk, any changes to this list should be addressed regularly with project management.

Please find below the link to the Issue Tracker document for more details

<Issue Tracker Document Link>

Open Items

Test Tools and Analysis

Test Execution

Load Runner <VERSION> will be used as the load testing .The following protocol(s) can be used for scripting .

Web

SOAP

Winsock

RTE

AMF

Additional protocols will be used per the application requirement.

Test Monitoring

SiteScope <Version Number > will be used for Performance Test Monitoring.

SiteScope counters will be configured to monitor the server (Web , App , DB ) components .

Additional tools like NMON , SAR will leveraged to have complete system logging and monitoring details during the test execution .

<Refer Test Monitoring Framework document>

Test Analysis

Performance Analysis will be done leveraging the information from the following

Load Runner exection details and logs

SiteScope Monitoring logs

Additional script level logging

Logs from the application interfaces(WAS , ESB etc) abd vendor party interfaces logs (FDR , ACBS , CMSI etc)

Appendix

<Additional documents >

<Reference documents>

Glossary

PCoE	Performance Center of Excellence	PCoE is an enterprise framework to establish and enable a strong quality processes. It comprises of strategic assets (people, process and technology) tied together to realize the vision of a zero defect organization. The PCoE framework leverages tools, techniques and methodologies to deliver technology better, faster and cheaper.
SLA	Service Level Agreement	This is the expected and agreed upon response of a business service of an application in production. SLAs commonly include segments to address: a definition of services, performance measurement, problem management, customer duties, warranties, disaster recovery, termination of agreement